Focusing on the information redundancy and encoding instability of depth image for gesture feature extraction in complex environment, an improved gesture feature extraction algorithm for depth image based on curvature-LBP (Local Binary Pattern) was proposed. Firstly, the divided gesture depth data to the point cloud data was converted through the coordinate conversion. Secondly, surface fitting was fulfilled with the moving least square method. And then the Gaussian curvature was calculated to describe the characteristics of the 3D surface geometry more accurately. Finally, the improved LBP uniform model was applied to encode the Gaussian curvature data and form a feature vector. In the American Sign Language (ASL) database, the average recognition rate of the proposed algorithm reached 91.20%, which 18.5 percentage points and 13.7 percentage points higher than 3DLBP and gradient LBP. Simulation results show that the proposed algorithm can recognize the gestures with similar outline and different shape, and improve the precision of describing the internal details in gesture depth image.